Computational Imaging for Shape Understanding

Geometry is the essential property of real-world scenes. Understanding the shape of the object is critical to many computer vision applications. In this dissertation, we explore using computational imaging approaches to recover the geometry of real-world scenes. Computational imaging is an emerging technique that uses the co-designs of image hardware and computational software to expand the capacity of traditional cameras. To tackle face recognition in the uncontrolled environment, we study 2D color image and 3D shape to deal with body movement and self-occlusion. Especially, we use multiple RGB-D cameras to fuse the varying pose and register the front face in a unified coordinate system. The deep color feature and geodesic distance feature have been used to complete face recognition. To handle the underwater image application, we study the angular-spatial encoding and polarization state encoding of light rays using computational imaging devices. Specifically, we use the light field camera to tackle the challenging problem of underwater 3D reconstruction. We leverage the angular sampling of the light field for robust depth estimation. We also develop a fast ray marching algorithm to improve the efficiency of the algorithm. To deal with arbitrary reflectance, we investigate polarimetric imaging and develop polarimetric Helmholtz stereopsis that uses reciprocal polarimetric image pairs for high-fidelity 3D surface reconstruction. We formulate new reciprocity and diffuse/specular polarimetric constraints to recover surface depths and normals using an optimization framework. To recover the 3D shape in the unknown and uncontrolled natural illumination, we use two circularly polarized spotlights to boost the polarization cues corrupted by the environment lighting, as well as to provide photometric cues. To mitigate the effect of uncontrolled environment light in photometric constraints, we estimate a lighting proxy map and iteratively refine the normal and lighting estimation. Through expensive experiments on the simulated and real images, we demonstrate that our proposed computational imaging methods outperform traditional imaging approaches

THEORETICAL RESEARCH ON EFFECTS OF SUBSTITUENTS AND THE SOLVENT ON QUADRUPLE HYDROGEN BONDED COMPLEXES

Semiempirical AM1 and INDO/CIS methods were used to study the structures and spectroscopy of hydrogen bonded complexes formed by the oligophenyleneethynylene (monomer A) with isophthalic acid (monomer B). The binding energies of the complexes are lowered by increasing electron-donating abilities of the substituents near the hydrogen bonds on monomer A. The first absorptions in the electronic spectra and the vibration frequencies of the N-H bonds in the IR spectra for the complexes are both red-shifted compared with those of the monomers. The presence of dimethylsulfoxide (DMSO) can reduce the binding energy of the complex through hydrogen bonding. This results in a blue-shift for the first absorption in the electronic spectrum and red-shift for the vibration frequencies of the N-H bonds in the IR spectrum of the complex. KEY WORDS: Oligophenyleneethynylene, Hydrogen bonding, Solvent effect, Semiempirical methods Bull. Chem. Soc. Ethiop. 2007, 21(3), 419-426

LGDN: Language-Guided Denoising Network for Video-Language Modeling

Video-language modeling has attracted much attention with the rapid growth of web videos. Most existing methods assume that the video frames and text description are semantically correlated, and focus on video-language modeling at video level. However, this hypothesis often fails for two reasons: (1) With the rich semantics of video contents, it is difficult to cover all frames with a single video-level description; (2) A raw video typically has noisy/meaningless information (e.g., scenery shot, transition or teaser). Although a number of recent works deploy attention mechanism to alleviate this problem, the irrelevant/noisy information still makes it very difficult to address. To overcome such challenge, we thus propose an efficient and effective model, termed Language-Guided Denoising Network (LGDN), for video-language modeling. Different from most existing methods that utilize all extracted video frames, LGDN dynamically filters out the misaligned or redundant frames under the language supervision and obtains only 2--4 salient frames per video for cross-modal token-level alignment. Extensive experiments on five public datasets show that our LGDN outperforms the state-of-the-arts by large margins. We also provide detailed ablation study to reveal the critical importance of solving the noise issue, in hope of inspiring future video-language work.Comment: Accepted by NeurIPS202

An Unified Search and Recommendation Foundation Model for Cold-Start Scenario

In modern commercial search engines and recommendation systems, data from multiple domains is available to jointly train the multi-domain model. Traditional methods train multi-domain models in the multi-task setting, with shared parameters to learn the similarity of multiple tasks, and task-specific parameters to learn the divergence of features, labels, and sample distributions of individual tasks. With the development of large language models, LLM can extract global domain-invariant text features that serve both search and recommendation tasks. We propose a novel framework called S\&R Multi-Domain Foundation, which uses LLM to extract domain invariant features, and Aspect Gating Fusion to merge the ID feature, domain invariant text features and task-specific heterogeneous sparse features to obtain the representations of query and item. Additionally, samples from multiple search and recommendation scenarios are trained jointly with Domain Adaptive Multi-Task module to obtain the multi-domain foundation model. We apply the S\&R Multi-Domain foundation model to cold start scenarios in the pretrain-finetune manner, which achieves better performance than other SOTA transfer learning methods. The S\&R Multi-Domain Foundation model has been successfully deployed in Alipay Mobile Application's online services, such as content query recommendation and service card recommendation, etc.Comment: CIKM 2023,6 page

Silver spoon effects of hatching order in an asynchronous hatching bird

The silver spoon hypothesis proposes that individuals which develop under favourable conditions will gain fitness benefits throughout their lifetime. Hatching order may create a considerable size hierarchy within a brood and lead to earlier-hatched nestlings having a competitive advantage over their siblings, which has been illustrated in some studies. However, there have been few explorations into the effect on subsequent generations. Here, using a 15-year-long study, we investigated the long-term fitness consequence of hatching order in the endangered crested ibis, Nipponia nippon, a species with complete hatching asynchrony. In this study, we found strong support for silver spoon effects acting on hatching order. Compared to later-hatched nestlings, first-hatched nestlings begin reproduction at an earlier age, have higher adult survival rates, possess a longer breeding life span and achieve higher lifetime reproductive success. Interestingly, we found carry-over effects of hatching order into the next generation. Nestlings which hatched earlier and became breeders in turn also produced nestlings with larger tarsus and better body condition. Additionally, we found a positive correlation among life-history traits in crested ibis. Individuals which started reproduction at a younger age were shown to possess a longer breeding life span. And the annual brood size increased with an individual’s breeding life span. This suggests that the earlier-hatched nestlings are of better quality and the ‘silver spoon’ effects of hatching order cover all life-history stages and next generation effects

Deep reinforcement learning for real-time economic energy management of microgrid system considering uncertainties

The electric power grid is changing from a traditional power system to a modern, smart, and integrated power system. Microgrids (MGs) play a vital role in combining distributed renewable energy resources (RESs) with traditional electric power systems. Intermittency, randomness, and volatility constitute the disadvantages of distributed RESs. MGs with high penetrations of renewable energy and random load demand cannot ignore these uncertainties, making it difficult to operate them effectively and economically. To realize the optimal scheduling of MGs, a real-time economic energy management strategy based on deep reinforcement learning (DRL) is proposed in this paper. Different from traditional model-based approaches, this strategy is learning based, and it has no requirements for an explicit model of uncertainty. Taking into account the uncertainties in RESs, load demand, and electricity prices, we formulate a Markov decision process for the real-time economic energy management problem of MGs. The objective is to minimize the daily operating cost of the system by scheduling controllable distributed generators and energy storage systems. In this paper, a deep deterministic policy gradient (DDPG) is introduced as a method for resolving the Markov decision process. The DDPG is a novel policy-based DRL approach with continuous state and action spaces. The DDPG is trained to learn the characteristics of uncertainties of the load, RES output, and electricity price using historical data from real power systems. The effectiveness of the proposed approach is validated through the designed simulation experiments. In the second experiment of our designed simulation, the proposed DRL method is compared to DQN, SAC, PPO, and MPC methods, and it is able to reduce the operating costs by 29.59%, 17.39%, 6.36%, and 9.55% on the June test set and 30.96%, 18.34%, 5.73%, and 10.16% on the November test set, respectively. The numerical results validate the practical value of the proposed DRL algorithm in addressing economic operation issues in MGs, as it demonstrates the algorithm’s ability to effectively leverage the energy storage system to reduce the operating costs across a range of scenarios